Effects of component-subscription network topology on large-scale data centre performance scaling

Modern large-scale date centres, such as those used for cloud computing service provision, are becoming ever-larger as the operators of those data centres seek to maximise the benefits from economies of scale. With these increases in size comes a growth in system complexity, which is usually problematic. There is an increased desire for automated "self-star" configuration, management, and failure-recovery of the data-centre infrastructure, but many traditional techniques scale much worse than linearly as the number of nodes to be managed increases. As the number of nodes in a median-sized data-centre looks set to increase by two or three orders of magnitude in coming decades, it seems reasonable to attempt to explore and understand the scaling properties of the data-centre middleware before such data-centres are constructed. In [1] we presented SPECI, a simulator that predicts aspects of large-scale data-centre middleware performance, concentrating on the influence of status changes such as policy updates or routine node failures. [...]. In [1] we used a first-approximation assumption that such subscriptions are distributed wholly at random across the data centre. In this present paper, we explore the effects of introducing more realistic constraints to the structure of the internal network of subscriptions. We contrast the original results [...] exploring the effects of making the data-centre's subscription network have a regular lattice-like structure, and also semi-random network structures resulting from parameterised network generation functions that create "small-world" and "scale-free" networks. We show that for distributed middleware topologies, the structure and distribution of tasks carried out in the data centre can significantly influence the performance overhead imposed by the middleware

Rosetta: Large scale system for text detection and recognition in images

In this paper we present a deployed, scalable optical character recognition (OCR) system, which we call Rosetta, designed to process images uploaded daily at Facebook scale. Sharing of image content has become one of the primary ways to communicate information among internet users within social networks such as Facebook and Instagram, and the understanding of such media, including its textual information, is of paramount importance to facilitate search and recommendation applications. We present modeling techniques for efficient detection and recognition of text in images and describe Rosetta's system architecture. We perform extensive evaluation of presented technologies, explain useful practical approaches to build an OCR system at scale, and provide insightful intuitions as to why and how certain components work based on the lessons learnt during the development and deployment of the system.Comment: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD) 2018, London, United Kingdo

Web Based Snow Sport Injury Reduction

Skiing and snowboarding injuries have plagued riders of all ages and skill levels. The website portion of this project aims to reduce these serious injuries through educating skiers and snowboarders over an Internet based platform. Inadvertent release is a release of the ski binding under circumstances that would normally not warrant a release. The inadvertent release film study aims to observe inadvertent release in skiers in order to determine the mechanisms that cause it. Our hypothesis for the fitness part of this project is that increased physical fitness reduces the risk of injury potentially injurious situations. The objective of the fitness study is to determine if it is possible to determine the risk of injury in skiing from physical fitness tests

Verifying atomicity via data independence

We present a technique for automatically verifying atomicity of composed concurrent operations. The main observation behind our approach is that many composed concurrent operations which oc-cur in practice are data-independent. That is, the control-flow of the composed operation does not depend on specific input values. While verifying data-independence is undecidable in the general case, we provide succint sufficient conditions that can be used to establish a composed operation as data-independent. We show that for the common case of concurrent maps, data-independence re-duces the hard problem of verifying linearizability to a verification problem that can be solved efficiently with a bounded number of keys and values. We implemented our approach in a tool called VINE and evalu-ated it on all composed operations from 57 real-world applications (112 composed operations). We show that many composed opera-tions (49 out of 112) are data-independent, and automatically verify 30 of them as linearizable and the rest 19 as having violations of linearizability that could be repaired and then subsequently auto-matically verified. Moreover, we show that the remaining 63 oper-ations are not linearizable, thus indicating that data independence does not limit the expressiveness of writing realistic linearizable composed operations. Categories and Subject Descriptors

GraphM : an efficient storage system for high throughput of concurrent graph processing

With the rapidly growing demand of graph processing in the real world, a large number of iterative graph processing jobs run concurrently on the same underlying graph. However, the storage engines of existing graph processing frameworks are mainly designed for running an individual job. Our studies show that they are inefficient when running concurrent jobs due to the redundant data storage and access overhead. To cope with this issue, we develop an efficient storage system, called GraphM. It can be integrated into the existing graph processing systems to efficiently support concurrent iterative graph processing jobs for higher throughput by fully exploiting the similarities of the data accesses between these concurrent jobs. GraphM regularizes the traversing order of the graph partitions for concurrent graph processing jobs by streaming the partitions into the main memory and the Last-Level Cache (LLC) in a common order, and then processes the related jobs concurrently in a novel fine-grained synchronization. In this way, the concurrent jobs share the same graph structure data in the LLC/memory and also the data accesses to the graph, so as to amortize the storage consumption and the data access overhead. To demonstrate the efficiency of GraphM, we plug it into state-of-the-art graph processing systems, including GridGraph, GraphChi, PowerGraph, and Chaos. Experiments results show that GraphM improves the throughput by 1.73~13 times

RPCValet: NI-Driven Tail-Aware Balancing of µs-Scale RPCs

Modern online services come with stringent quality requirements in terms of response time tail latency. Because of their decomposition into fine-grained communicating software layers, a single user request fans out into a plethora of short, μs-scale RPCs, aggravating the need for faster inter-server communication. In reaction to that need, we are witnessing a technological transition characterized by the emergence of hardware-terminated user-level protocols (e.g., InfiniBand/RDMA) and new architectures with fully integrated Network Interfaces (NIs). Such architectures offer a unique opportunity for a new NI-driven approach to balancing RPCs among the cores of manycore server CPUs, yielding major tail latency improvements for μs-scale RPCs. We introduce RPCValet, an NI-driven RPC load-balancing design for architectures with hardware-terminated protocols and integrated NIs, that delivers near-optimal tail latency. RPCValet's RPC dispatch decisions emulate the theoretically optimal single-queue system, without incurring synchronization overheads currently associated with single-queue implementations. Our design improves throughput under tight tail latency goals by up to 1.4x, and reduces tail latency before saturation by up to 4x for RPCs with μs-scale service times, as compared to current systems with hardware support for RPC load distribution. RPCValet performs within 15% of the theoretically optimal single-queue system

Hermes: a Fast, Fault-Tolerant and Linearizable Replication Protocol

Today's datacenter applications are underpinned by datastores that are responsible for providing availability, consistency, and performance. For high availability in the presence of failures, these datastores replicate data across several nodes. This is accomplished with the help of a reliable replication protocol that is responsible for maintaining the replicas strongly-consistent even when faults occur. Strong consistency is preferred to weaker consistency models that cannot guarantee an intuitive behavior for the clients. Furthermore, to accommodate high demand at real-time latencies, datastores must deliver high throughput and low latency. This work introduces Hermes, a broadcast-based reliable replication protocol for in-memory datastores that provides both high throughput and low latency by enabling local reads and fully-concurrent fast writes at all replicas. Hermes couples logical timestamps with cache-coherence-inspired invalidations to guarantee linearizability, avoid write serialization at a centralized ordering point, resolve write conflicts locally at each replica (hence ensuring that writes never abort) and provide fault-tolerance via replayable writes. Our implementation of Hermes over an RDMA-enabled reliable datastore with five replicas shows that Hermes consistently achieves higher throughput than state-of-the-art RDMA-based reliable protocols (ZAB and CRAQ) across all write ratios while also significantly reducing tail latency. At 5% writes, the tail latency of Hermes is 3.6X lower than that of CRAQ and ZAB.Comment: Accepted in ASPLOS 202

Learning together for and with the Martuwarra Fitzroy River

Co-production across scientific and Indigenous knowledge systems has become a cornerstone of research to enhance knowledge, practice, ethics, and foster sustainability transformations. However, the profound differences in world views and the complex and contested histories of nation-state colonisation on Indigenous territories, highlight both opportunities and risks for Indigenous people when engaging with knowledge co-production. This paper investigates the conditions under which knowledge co-production can lead to improved Indigenous adaptive environmental planning and management among remote land-attached Indigenous peoples through a case study with ten Traditional Owner groups in the Martuwarra (Fitzroy River) Catchment in Western Australia’s Kimberley region. The research team built a 3D map of the river and used it, together with an interactive table-top projector, to bring together both scientific and Indigenous spatial knowledge. Participatory influence mapping, aligned with Traditional Owner priorities to achieve cultural governance and management planning goals set out in the Fitzroy River Declaration, investigated power relations. An analytical framework, examining underlying mechanisms of social learning, knowledge promotion and enhancing influence, based on different theories of change, was applied to unpack the immediate outcomes from these activities. The analysis identified that knowledge co-production activities improved the accessibility of the knowledge, the experiences of the knowledge users, strengthened collective identity and partnerships, and strengthened Indigenous-led institutions. The focus on cultural governance and management planning goals in the Fitzroy River Declaration enabled the activities to directly affect key drivers of Indigenous adaptive environmental planning and management—the Indigenous-led institutions. The nation-state arrangements also gave some support to local learning and decision-making through a key Indigenous institution, Martuwarra Fitzroy River Council. Knowledge co-production with remote land-attached Indigenous peoples can improve adaptive environmental planning and management where it fosters learning together, is grounded in the Indigenous-led institutions and addresses their priorities